The present invention relates to a process for integrating results of works performed by multiple workers, and more particularly to an estimation method, an estimation system, a computer system and a program for estimating abilities of multiple workers in a process for integrating results of works performed by the workers.
Recently, attention has been paid to crowdsourcing in which works are entrusted to an unspecified large number of workers. In crowdsourcing, the same task is entrusted to a lot of workers, work results are integrated, and an integrated work result of the task is obtained. It is possible to obtain the work result with a higher quality by performing the integration appropriately.
In the case of attempting to integrate answers from a lot of workers to obtain a correct answer, a simple method is thought of, which uses majority voting to determine the correct answer. In the case of simple majority voting, however, differences among correct answer rates of the workers is ignored. Therefore, there is proposed a technique for improving the accuracy of the obtained answer by estimating skills of the workers (for example, correct answer rates) and the degree of difficulty of works and, thereby, performing weighted integration.
In the conventional technique mentioned above, however, it is assumed that individual workers create answers from the beginning. Therefore, the technique is not sufficiently compatible with a step-by-step workflow in which succeeding-stage workers work based on a work result of a preceding-stage worker. In one example, modification may be performed based on a result of Automatic Speech Recognition (ASR) for work of voice or video transcription. In another example, a workflow includes a person modifying a result of Optical Character Recognition (OCR) for work of digitization of a book.
In the above workflows, there is a possibility that succeeding-stage workers are influenced by seeing the answer of a preceding-stage worker. For example, there is a possibility that a worker exists who has a tendency of believing an automatic recognition result and not modifying wrong recognition, and a worker who has, on the contrary, a tendency of being suspicious and excessively modifying even correct recognition.
From the background described above, there has been a demand for development of a technique capable of, in a workflow in which succeeding-stage workers work based on a work result of a preceding-stage worker for the same task, and work results of these workers are integrated, estimating the behaviors and skills of the workers more accurately and performing integration with higher accuracy.